1. Field of the Invention
The present invention relates to an environment monitoring system for picking up the front, rear or diagonal-rear view from one""s own vehicle by an image pick-up means such as a video camera installed on the vehicle such as a motor car, detecting another vehicle approaching from the front, rear or diagonal-rear direction of one""s own running vehicle using the image picked up and giving a warning to a driver.
2. Description of the Related Art
For example, when a driver of a vehicle running on one-side two lanes of e.g. a speed way intends to change his own vehicle lane, if he changes the lane while he misses another vehicle which catches up with his own vehicle on another lane at a higher speed than his own vehicle from the diagonal-rear direction, there is strong possibility of a serious accident. Therefore, it is desired that the driver surely notices or recognizes other vehicles in the environment.
When another following vehicle on the same lane abruptly approaches his own vehicle from the rear, if the driver of his own vehicle makes abrupt braking, there is possibility of bumping-into-the back.
When another vehicle runs forward on the same lane at a lower speed than that of his own vehicle, if a driver dozes, there is a danger of bumping-into-the back.
Particularly, in recent years, various types of vehicles inclusive of a passenger can have a highly improved performance so that the running speed or acceleration thereof has been greatly increased. Therefore, the vehicle moves very fast in lane changing or acceleration/deceleration. As a result, when the driver misses the presence of another vehicle running in the environment of his own vehicle, there has been increased strong possibility of occurrence of a dangerous state which may lead to serious accidents, or an actual serious accident.
In order to prevent the driver from missing another in the environment than his own vehicle (hereinafter referred to as xe2x80x9csurrounding vehiclexe2x80x9d), the driver himself must first pay attention to the environment. However, it is true that attention or recognition ability of a human being has reached the limit of ability of capable of following the improved performance of a vehicle. Actually, there is an increased tendency of accidents attributable to the missing the surrounding vehicle at the limit value.
Generally, it is known that the visual recognition ability of a human being abruptly decreases as the running speed increases. The vehicle is still desired to deal with a higher running speed in the future. Therefore, there is a serious problem of danger of missing the presence of the surrounding vehicle in e.g. lane changing.
Accordingly, in order to prevent the danger of a great accident or the actual great accident from occurring, only enhancing the attention of the driver to the presence of the surrounding vehicle is not sufficient. It is necessary to recognize the presence of the surrounding vehicle running in the environment automatically, accurately and surely and give the acquired information to the driver so that the limited recognition ability of a human being can be complemented.
Several techniques for avoiding the danger as described above have been proposed as disclosed in JP-A-6-107096 and JP-A-7-50769. Specifically, JP-A-6-107096 discloses a system for detecting an approaching vehicle and collision warning based on the detection of an optical flow. JP-A-7-50769 discloses a warning system which intends to detect an approaching vehicle from the diagonal-rear direction on the basis of detection of an optical flow and issues a warning when there is a danger of colliding with another vehicle in lane changing.
Now referring to FIGS. 20 and 21, an explanation will be given of the summary of a conventional environment monitoring system.
FIGS. 20A-20D are views for explaining a change in a rear background image acquired by a video camera 1. FIG. 20A shows a status inclusive of one""s own vehicle. FIG. 20B shows an image picked up by a video camera 1 at timing t in an environment of one""s own vehicle. FIG. 20C shows an image picked up at timing t+xcex94t.
Now it is assumed that one""s own vehicle is running straight on a flat road. The road sign and building residing in the rear of one""s own vehicle in FIG. 20A are observed as images shown in FIGS. 20B and 20C at timings t and t+xcex94t, respectively. Connecting the corresponding points in these two images provides speed vectors as shown in FIG. 20D. The are referred to as xe2x80x9coptical flowsxe2x80x9d. Where a following vehicle approaches, the directions of the vectors in the optical flows in FIG. 20D are contrary.
These optical flows appear radially from a point called xe2x80x9cFOExe2x80x9d (Focus of Expansion) in the image. The FOE, which refers to xe2x80x9cinfinite pointxe2x80x9d or xe2x80x9cdisappearing pointxe2x80x9d, corresponds to the one point opposite to the running direction of one""s own vehicle on the image where one""s own vehicle runs straight. In this way, the optical flows acquired when one""s own vehicle runs extends radially around the FOE. The optical flows issued from the following vehicle or another vehicle running on an adjacent lane (hereinafter referred to as xe2x80x9cparticular vehiclexe2x80x9d) include information composed of a position and relative speed of the above particular vehicle. Where the optical flows are longer and diverge from the FOE, there is strong possibility of danger.
Now referring to FIG. 21, a detailed explanation will be given of the optical flows. In an optical arrangement shown in FIG. 21, it is assumed that reference numeral 11 denotes a lens of a video camera, 12 denotes an image plane of the video camera, f denotes a distance from the lens 11 to the image plane 12, P (X, Y, Z) denotes any point on the following vehicle, and p (x y) denotes a point corresponding to the point P on the image plane 12.
From the ratio of similarity of a triangle,
x=fxc2x7X/Zxe2x80x83xe2x80x83(1)
By transforming and time-differentiating this equation,
xxe2x80x2=(xcex94x/xcex94txc2x7Z+xxc2x7Zxe2x80x2)/fxe2x80x83xe2x80x83(2)
The component u in the x direction of the optical flow is expressed by
u=xcex94x/xcex94txe2x80x83xe2x80x83(3)
Using it, Z can be expressed by
Z=(fxc2x7Xxe2x80x2xe2x88x92xxc2x7Zxe2x80x2)/uxe2x80x83xe2x80x83(4)
Now, Z denotes a relative speed between one""s own vehicle and the xe2x80x9cparticular vehiclexe2x80x9d. Assuming that
Zxe2x80x2=xe2x88x92xcex1,xe2x80x83xe2x80x83(5)
Equation (4) can be transformed into
Z=(fxc2x7Xxe2x80x2+x xcex1)/uxe2x80x83xe2x80x83(6)
Therefore, the x direction component of the optical flow (i.e. xcex94x/xcex94t=u) can be expressed by
u=(fxc2x7Xxe2x80x2+xxcex1)/Zxe2x80x83xe2x80x83(7)
This applies to Y.
Thus, from Equation (7), as Z i.e. the distance from the particular vehicle is decreased, otherwise xcex1 is increased (relative speed is large), the x component of the optical flow is increased. This applies to the Y direction.
Therefore, the optical flow is longer as the distance from the xe2x80x9cparticular vehiclexe2x80x9d is shorter or the relative speed is higher. Accordingly, a greater length of the optical flow diverging from FOE provides higher degree of danger for the particular vehicle.
The above processing is repeated for all the points on the image at timing t so that the optical flows of the entire image representative of the danger of each particular vehicle can be acquired. An alarm is sounded according to the acquired degree of danger to alert the driver of his own vehicle to the danger. This complements the limited recognition ability of a human being, and prevents the danger of a great accident or the actual occurrence thereof.
In a prior art, as shown in FIG. 22, in order to save the processing time for an object for which monitoring is not required, the white lines of a lane on which one""s own vehicle runs on a straight road are detected so that the lane on which one""s own vehicle runs is distinguished from an adjacent lane to define a monitoring region. The FOE point is acquired from the extension of the detected white line, and the optical flows extending radially from the FOE point are acquired for one""s own vehicle lane and adjacent lane, thereby detecting a particular vehicle 102 approaching one""s own vehicle. In this way, the particular vehicle is recognized on the basis of the optical flows so that the degree of danger for the particular vehicle can be automatically decided without using any particular distance meter.
The optical flow represents a moving vector on an image which is produced due to a relative movement between a camera and an object in a three-dimensional space. Of the techniques for detecting the optical flows which have been proposed hitherto, a correlation technique and a concentration gradient technique can be practically adopted at present.
The moving image picked up from one""s own vehicle mixedly includes an optical flow occurring from background approaching at a very high speed and optical flow occurring from a forward vehicle approaching or going away at various speeds. The moving image is characterized in that it is changed complicatedly with elapse of time. Therefore, it cannot be assumed that changes in luminance are smooth in time and space, which is required by the concentration gradient technique, so that the optical flow cannot be detected accurately. Accordingly, the concentration gradient technique is unsuitable for the moving image to which the present invention is directed.
On the other hand, the correlation technique, which must search the corresponding points in a prescribed window in all the directions in environmental regions to compute the correlative values, requires a vast amount of computations disadvantageously. The correlation technique, however, has an advantage that it can acquire the optical flow relatively accurately for a complicate image to which the present invention is directed.
As described above, in the general correlation technique, in order to acquire the optical flow of the image at a certain timing t=T, the pixels at timing t=Txe2x88x92xcex94T corresponding to all the pixels at timing t=T in all the directions must be searched. This requires a vast amount of computations, which may lead to search of erroneous corresponding pixels (erroneous corresponding).
The problem of detecting the wall of a road or preceding vehicle outside the monitoring region can be solved by setting a monitoring region. However, even when the corresponding pixels are searched in only the set monitoring region still, problems of processing time and detecting accuracy are not still solved.
Meanwhile, searching of the corresponding points can be made accurately for only the pixels each having a characterized texture. For example, where the corresponding points are searched between the two images picked up with a time difference of xcex94T for a texture having a uniform concentration distribution such as an asphalt road, all points on the road are candidates of the corresponding points. This leads to the increase in the amount of computation and erroneous corresponding. Therefore, in order to search the corresponding points accurately, the pixels each having a characterized texture must be extracted.
The most simple technique for extracting the pixels each having a characteristic texture is to extract the edge of an image (which refers to the portion where the brightness varies on the image). However, when the edge of each image is extracted between the images picked up with a difference of xcex94T, the character on the road (e.g. painted character such as xe2x80x9cSTOPxe2x80x9d) which should not be detected as an optical flow will be extracted as an xe2x80x9cedgexe2x80x9d. This leads to an increase in the amount of computations.
Further, where a zebra zone, a pattern or character of speed limitation, etc. are painted on the road surface, optical flows will be produced from these paints. Essentially, it is intended to detect only the optical flow from an approaching vehicle. In this case, however, the optical flows from the paints may be erroneously detected as those from the approaching vehicle.
The present invention has been accomplished in order to accomplish the problems described above.
An object of the present invention is to provide an environment monitoring system and method which can prevent an object (e.g. paint on the road surface) which is not required to detect from being erroneously detected as another surrounding vehicle.
In order to attain the above object, in accordance with the first aspect of the present invention, there is provided an environment monitoring system for monitoring a relative relation between one""s own vehicle and another surrounding vehicle comprising: image pick-up means for picking up two early and later images which are located at a front, rear or diagonal-rear position from one""s own vehicle at two early and later timings and on the basis of the images thus picked-up, detecting white lines of one""s own vehicle lane on which one""s own vehicle runs; monitoring region setting means for setting a monitoring region in a region of one""s own vehicle lane or an adjacent lane on the basis of the white lines thus detected; optical-flow detecting means for detecting an optical flow generated from another surrounding vehicle within the monitoring region, wherein the optical-flow detecting means comprises: means for reverse-projection converting the early image on the basis of a predetermined optical arrangement of the image pick-up means onto an x-z plane in parallel to a road surface in a real space to acquire a road surface image; means for computing a moving distance of one""s own vehicle between the two timings on the basis of a time interval between the two timings and speed information of one""s own vehicle; means for parallel-shifting the road surface image by the moving distance thus computed; means for projection-converting the road surface image after parallel-shifted to acquire an estimated image of the later image in timing; means for acquiring a differential image between the later image and its estimated image to extract a feature point; and means for searching a corresponding point of the feature point extracted, thereby detecting the optical flow.
In this configuration, shadow, character and dirt whose optical flow should not be detected can be entirely canceled, and the corresponding point of only the object whose optical flow should be detected can be searched.
Preferably, the optical-flow detecting means further comprises: means for, prior to projection-converting the road surface image, computing angles of the white lines in the vicinity of one""s own vehicle formed together with a z-axis on the road surface image parallel-shifted to acquire an average angle thereof; and means for rotating the road surface by the average angle around an origin of the x-z coordinate.
In this configuration, an estimated image can be acquired taking a curved road into consideration. Thus, the shadow, character and dirt whose optical flow should not be detected can be entirely canceled. Therefore, when one""s own vehicle runs on the curved road, the corresponding point of only the object whose optical flow should be detected can be searched.
Preferably, the environment monitoring system further comprises means for evaluating the degree of danger which decides dangerous when the average value 1 of the magnitudes of optical flows exceeds a prescribed value as regards the degree D of danger in a front, rear or diagonal-rear field of view expressed by   D  =      l    =                  (                  1          /          N                )            xc3x97                        ∑                      i            =            1                    N                ⁢                  xe2x80x83                ⁢        li            
(where li denotes the magnitude of the optical flow, and N denotes the number of generated optical flows).
In this configuration, irrespectively of the generated number of optical flows which depends on the type, color, size of another environmental vehicle, and environmental brightness in the daytime or nighttime, the degree of danger can be evaluated.
Preferably, the means for evaluating the degree of danger decides dangerous when the number N of the optical flows exceeds a prescribed value. Even when the relative speed between one""s own vehicle and another surrounding vehicle is zero, if the distance between one""s own vehicle and forward vehicle is very short, the degree of danger becomes high. If the distance between the vehicles is short, the generated number N of optical flows increases irrespectively of the type, color and size of the vehicle. In this configuration, using this fact, the degree of danger can be evaluated.
In accordance with the second aspect of the present invention, there is provided a method for monitoring a relative relation between one""s own vehicle and another surrounding vehicle comprising the steps of: picking up two early and later images which are located at a front, rear or diagonal-rear position from an own vehicle at two early and later timings by an image pick-up means and on the basis of the images thus picked-up, detecting a white line of an own vehicle lane on which one""s own vehicle runs; setting a monitoring region in a region of one""s own vehicle lane or an adjacent lane on the basis of the white line thus detected; detecting an optical flow generated from another environmental vehicle within the monitoring region, wherein the step of detecting an optical flow comprises the steps of: reverse-projection converting the early image on the basis of a predetermined optical arrangement of the image pick-up means onto an x-z plane in parallel to a road surface in a real space to acquire a road surface image; computing a moving distance of one""s own vehicle between the two timings on the basis of a time interval between the two timings and speed information of one""s own vehicle; parallel-shifting the road surface image by the moving distance thus computed; projection-converting the road surface image after parallel-shifted to acquire an estimated image of the later image in timing; acquiring a differential image between the later image and its estimated image to extract a feature point; and searching a corresponding point of the feature point extracted, thereby detecting the optical flow.
In this configuration, shadow, character and dirt whose optical flow should not be detected can be entirely canceled, and the corresponding point of only the object whose optical flow should be detected can be searched.
Preferably, in the method according to the second aspect of the present invention, the step of detecting an optical-flow further comprises:
if prior to projection-converting the road surface image, computing angles of the white lines in the vicinity of one""s own vehicle formed together with a z-axis on the road surface image parallel-shifted to acquire an average angle thereof; and
rotating the road surface by the average angle around an origin of the x-z coordinate.
In this manner, an estimated image can be acquired taking a curved road into consideration. Thus, the shadow, character and dirt whose optical flow should not be detected can be entirely canceled. Therefore, when one""s own vehicle runs on the curved road, the corresponding point of only the object whose optical flow should be detected can be searched.
The above and other objects and features of the present invention will be more apparent from the following description taken in conjunction with the accompanying drawings.